Rebuild keyboard#1403
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
IDisposable
force-pushed
the
feat/better-keyboard
branch
2 times, most recently
from
4824b57 to
d209891
Compare
|
This doesn't have #1399 (which is unmerged), it will be easier for me to pull in his PR's content to this if we want those changes since I changed a lot of the keyboard pathway. |
|
Once we're in sync on this, I'll build the e2e and run the machine-translate. |
adamshiervani
left a comment
adamshiervani
left a comment
There was a problem hiding this comment.
This is the culmination of a lot of ideas you've had about the keyboard layout parsing - amazing to see all of this coming together!!
The overall direction of moving keyboard layout handling to a KLE-backed flow is great, but I’d like to tighten the scope of this PR before merging. Other than the PR comments:
- Remove custom keyboard layout uploads - I don’t think uploads will happen often enough to justify a permanent upload/storage/delete path in the product. If someone needs a new layout, we can add it as a built-in layout and then it works reliably for everyone going forward.
- Numpad Return key is overflowing the virtual keyboard - It looks like the Big Ass numpad Enter key is overflowing.
- Easier way for users to add layouts - Most people just want to add support for the most common layout of for their language - keyboard-layout-editor is too complex for most people
Lastly, also note, we don't have to keep backwards compatibility. The cloud is now versioned, and the device served JS always matches the backend.
I am not sure why we would want to not offer configurable keyboards, for example we let them upload ISO images. These are super tiny files. |
Turns out that what I ensure is that we use/upload any valid keyboard from https://kbdlayout.info since those are already built for just about every language imaginable. They just have to assign it a name and a code. |
Oops, a problem in the |
IDisposable
force-pushed
the
feat/better-keyboard
branch
from
026bdfa to
19c9b2c
Compare
|
@adamshiervani I think this is GTG now... I did a complete audit of the keyboard layouts, fixed a ton of issues, and cleaned up the rendering when the keyboard changes size (and also the detached keyboard is now resizable). We should accept any valid KLE file from https://kbdlayout.info ... just find a keyboard you want, click the KLE json file link and you get the baseline data. Add the header for deadkeys, name, etc.. and upload. 
|
IDisposable
force-pushed
the
feat/better-keyboard
branch
from
45b5b7f to
f4be415
Compare
|
@adamshiervani now I think we're just down to the machine-translate. Everything else seems to be working right and I've manually tested essentially every key on every layout :) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
The existing DEVELOPMENT.md "Adding a New Built-in Layout" section was written as a quick reference for someone already familiar with the codebase. New contributors trying to add a layout from scratch wanted something more like a tutorial — what kbdlayout.info is, how to clone an existing layout, what each metadata field means, how legends map to keycap corners, when to use deadKeys vs not, what the audit script is for, what to expect in the UI. This is that tutorial. Mermaid diagrams cover the data flow, the two recommended starting paths (clone vs kbdlayout.info), and the paste charMap pipeline. ASCII keycap mockups show the four-corner layer layout. Naming conventions, aliases, troubleshooting, and the metadata reference are all in one place. DEVELOPMENT.md keeps a brief "Adding a Layout (Quick Reference)" section that points to the new tutorial. DESIGN.md and TRANSPORT.md get a single-line callout at the top so contributors landing there first get redirected.
The template was written before the contributor walkthrough existed.
Now that ADDING_A_LAYOUT.md covers the full PR path, restructure the
issue template to:
- Lead with "open a PR if you can — see ADDING_A_LAYOUT.md."
- Position the issue itself as the fallback for non-coders or layout
bug reports.
- Make a kbdlayout.info URL the preferred input (most reliable for
us, lets the audit script validate against the canonical reference)
and the KLE JSON the alternative.
- Add a required "dead keys" field with explicit guidance on how to
verify a key is actually a dead key on the target OS — getting this
wrong is the most common bug source on existing layouts.
- Update the display-name guidance with the convention used by the
built-in layouts ("native + locale + form-factor", e.g.
"한국어 ko-KR (ANSI 103)").
- Update the locale-code guidance to match what the codebase wants
(`language-REGION` with hyphen, hyphenated in the ID, underscored
in the filename — covered in detail in the doc).
- Refresh the checklist to match the new doc and to make the
"validate locally" step optional rather than required (many issue
submitters don't have Go installed; that's why they're filing an
issue).
Runs whenever a PR or push touches keyboard parser/layout/script files.
Path-filtered so unrelated PRs don't pay for it.
Two checks per run:
1. go test ./internal/keyboard/... — parser, charMap, drift-guard
(TestBuiltinLayoutLegendsAreKnown), dead-key compositions,
control-scancode contract.
2. go run scripts/audit_layouts.go — compares each built-in layout
against its kbdlayout.info reference. Exits non-zero only on
real regressions (missing charMap entries); known ISO/ANSI and
dead-key-indirection differences are emitted as warnings and
don't fail the job.
Mechanics:
- actions/cache persists ~/.cache/kbdlayout-audit between runs, keyed
by the hash of the layout files. First run on a new layout downloads
the reference; subsequent runs hit the cache.
- Audit output is appended to the GITHUB_STEP_SUMMARY (last 25 lines
show up directly on the PR check page) and uploaded as an artifact
for full inspection.
- Path filter covers internal/keyboard/**, scripts/audit_layouts.go,
scripts/validate_layout.go, and the workflow file itself, so any
changes to the validation tooling re-trigger it.
Three pieces in CLAUDE.md were really general developer wisdom mis-
filed as AI-only guidance — useful to anyone touching the codebase.
Promote them to the contributor-facing dev guide:
- The 3-mode supervisor binary in cmd/main.go (parent supervisor,
main app, native subprocess). Useful when debugging crashes or
touching cmd/main.go.
- jsonrpc.go as the central RPC dispatch — added to the "Key files
for beginners" list, where it actually gets edited more often than
web.go for new functionality.
- Internal packages purpose table — DEVELOPMENT.md already had a
project-tree but no per-package blurbs. Pasted from CLAUDE.md and
expanded slightly (added confparser, diagnostics, mdns, timesync,
tzdata, utils which were missing).
- Linting rules cheat sheet — golangci forbidigo / gochecknoinits
rules and the oxlint hook details, so contributors know about
them before the pre-commit hook fails. Includes the "use String(e)
not interpolation" gotcha that bit recently.
- PR target = dev branch. Said out loud since the Makefile already
enforces it but new contributors might not see that.
CLAUDE.md keeps the AI-specific framing and a few items genuinely
specific to working with the assistant; the new DEVELOPMENT.md
sections supersede the duplicate content.
Replace the catch-all 'internal/keyboard/**' glob with one entry per intent so a reader doesn't have to know that '**' is recursive: - layouts/** — the built-in KLE JSON files - testdata/** — KLE fixtures the unit tests use - keyaliases.json — the special-key taxonomy - **.go — parser, validator, tests, helpers Functionally equivalent to the previous recursive glob (verified via git ls-files against each pattern); the value is readability when someone scans the workflow to see what it cares about.
|
One question... there are keyboards like Canadian Multilingual Standard that use Right-Ctrl (aka OEM 8) and RCtrl+Shift for more than the normal legend slots... for example @adamshiervani do you want me to add that support before finalizing the doc and shipping this, or do you want to follow one with another PR? 
|
…orical framing
The keyboard work on this branch reached a clean, shippable state, but
the docs had drifted in three ways:
(1) Field-name mismatches with the actual transport schema.
- TransportKey ships `deadLegends` (a list of legend slot names) and
NOT a boolean `dead` flag. DESIGN.md and TRANSPORT.md both still
described the old shape. TRANSPORT.md's example block had a literal
"dead": false field that does not exist on the wire.
- TransportKey ships `controlLike` (a Go-computed boolean shipped
with each key — see prior commit f9d6a40). The TRANSPORT.md
section on it was kept but its "earlier helpers misclassified X"
framing is removed; doc now describes the current state directly.
(2) JSON-RPC method list was incomplete.
TRANSPORT.md listed getKeyboardLayouts / getKeyboardLayoutData /
deleteKeyboardLayout but omitted getKeyboardLayout (returns active
ID) and setKeyboardLayout (persists active ID). Both are first-class
RPCs in jsonrpc.go and the layout settings UI uses them.
(3) "Previously" / "now" framing.
Per the maintainer note that none of the keyboard work has shipped,
every doc passage that compared the new state to the old state is
noise to a reader who only sees the result. Pruned:
- DESIGN.md "Background" section's bullet list of what the system
"was English-only" / "now provide[s]" became a description of the
three concerns the system covers today.
- ADDING_A_LAYOUT.md troubleshooting row about an "old build" aria
label format is gone.
- TRANSPORT.md "Scancode classification" section dropped its
explanation of which earlier helper had a bug; describes only the
current functions.
(4) Doc cross-references caught up.
- DESIGN.md's File Reference block now mentions ADDING_A_LAYOUT.md
and the new keyaliases.{go,json} pair under internal/keyboard/.
- DESIGN.md's aria-label note now points to the keyaliases.json
taxonomy as the single source of truth shared with Go.
- TRANSPORT.md's POST /keyboard/upload doc gained the ?id= query
parameter (used to replace an existing user-uploaded layout —
already supported by the handler).
Spot-checked DEVELOPMENT.md (root) and docs/keyboard/DEVELOPMENT.md
in the same pass — both already describe current state cleanly.
Add ripgrep to the devcontainer Everything current except Zustand and xterm.
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
IDisposable
force-pushed
the
feat/better-keyboard
branch
from
515f8d9 to
8dfd701
Compare
IDisposable
force-pushed
the
feat/better-keyboard
branch
from
8dfd701 to
0c94d2b
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 24c1a09. Configure here.
adamshiervani
left a comment
adamshiervani
left a comment
There was a problem hiding this comment.
We should accept any valid KLE file from kbdlayout.info ... just find a keyboard you want, click the KLE json file link and you get the baseline data. Add the header for deadkeys, name, etc.. and upload.
I still couldn't get that to work
| // Handle form submission. FormData entries are FormDataEntryValue (string|File); | ||
| // these three are always strings, but narrow explicitly so url interpolation | ||
| // can't accidentally stringify a File. | ||
| const formData = await request.formData(); | ||
| const stringField = (k: string): string => { | ||
| const v = formData.get(k); | ||
| return typeof v === "string" ? v : ""; | ||
| }; | ||
| const name = stringField("name"); | ||
| const id = stringField("id"); | ||
| const returnTo = stringField("returnTo"); |
There was a problem hiding this comment.
LLM went to the sidelines here?
There was a problem hiding this comment.
That does seem like an odd dance... I'll simplify.
| const { deviceId } = Object.fromEntries(await request.formData()); | ||
| const formData = await request.formData(); | ||
| const rawDeviceId = formData.get("deviceId"); | ||
| const deviceId = typeof rawDeviceId === "string" ? rawDeviceId : ""; |
| } | ||
| }); | ||
| }); | ||
|
|
There was a problem hiding this comment.
All these look like UI tests and not remote agent based
There was a problem hiding this comment.
I don't have anything to run the remote agent on, so I never worked that part through. I've got the test JetKVM hooked up to my Proxmox, so not sure the process.
| await goToSession(page); | ||
| }); | ||
|
|
||
| test("clicking a virtual key sends the correct scancode", async ({ page }) => { |
There was a problem hiding this comment.
Use remote agent for this to truly verify the scan code instead of the caps lock hack we used before
| @@ -0,0 +1,188 @@ | |||
| /** | |||
| * LayoutPreviewDialog — modal that shows a keyboard layout preview. | |||
There was a problem hiding this comment.
Still used when they upload one, just not used in the selector anymore.
| onChange={onKeyboardLayoutChange} | ||
| options={keyboardOptions} | ||
| /> | ||
| <Listbox value={keyboardLayout} onChange={onKeyboardLayoutChange}> |
There was a problem hiding this comment.
Use <SelectMenuBasic /> now that we only use strings as labels
2 participants
fr-BElayout +nl-BEaliashu-HU+ GitHub issue templateSummary
The key concept is the target layout — the keyboard layout configured on
the controlled machine. The physical keyboard passthrough uses HID scancodes
(position-based, like a USB cable), which is correct KVM behaviour. But the
virtual keyboard, paste text, and macro display all need to know the target
layout to show correct legends and send correct character sequences.
The virtual on-screen keyboard needs to:
Checklist
make test_e2elocally and passedCloses #<issue-number>)Changes
Before
Keyboard Selector
Virtual Keyboard
After
Keyboard Selector
Virtual Keyboard
Quick Keys
Preview in keyboard selector
Sticky keys in Virtual Keyboard is optional
Note
High Risk
Introduces a new keyboard-layout parsing/storage pipeline and related RPC/HTTP handlers, which can affect core input behavior and adds a new user-supplied JSON parsing surface. Also changes CI/dev tooling and e2e coverage, so regressions could impact development and release validation.
Overview
Adds a new KLE (keyboard-layout-editor) driven keyboard layout system on the Go backend (
internal/keyboard), including embedded built-in layouts discovery/aliasing, charMap generation (with dead-key composition), and control-key legend normalization via a sharedkeyaliases.jsontaxonomy.Introduces layout management endpoints: HTTP upload (
POST /keyboard/upload) storing processed layouts under/userdata/kvm_layouts/, plus JSON-RPC handlers to list, fetch, and delete layouts with caching and anen-USfallback.Build/test workflow updates: new GitHub Action
keyboard.ymlruns keyboard unit tests +kbdlayout.infoaudit (with caching/artifacts); Makefile e2e suite now includeskeyboard-paste/keyboard-macrosPlaywright projects; devcontainers install Playwright deps + Docker CLI and mount~/.gnupg; adds a keyboard-layout issue template and updates lint/gitignore/docs accordingly.Reviewed by Cursor Bugbot for commit 24c1a09. Bugbot is set up for automated code reviews on this repo. Configure here.